We had
some requests to get
GHC (the Glasgow Haskell Compiler) up and running on two new Ubuntu architectures:
arm64, added in 13.10, and
ppc64el, added in 14.04. This has been something of a saga, and has involved rather more late-night hacking than is probably good for me.
Book the First: Recalled to a life of strange build systems
You might not know it from the sheer bulk of uploads I do sometimes, but I actually don't speak a word of Haskell and it's not very high up my list of things to learn. But I am a pretty experienced build engineer, and I enjoy porting things to new architectures: I'm firmly of the belief that breadth of architecture support is a good way to shake out certain categories of issues in code, that it's worth doing aggressively across an entire distribution, and that, even if you don't think you need something now, new requirements have a habit of coming along when you least expect them and you might as well be prepared in advance. Furthermore, it annoys me when we have excessive noise in our
build failure and
proposed-migration output and I often put bits and pieces of spare time into gardening miscellaneous problems there, and at one point there was a lot of Haskell stuff on the list and it got a bit annoying to have to keep sending patches rather than just fixing things myself, and ... well, I ended up as probably the only non-Haskell-programmer on the Debian Haskell team and found myself fixing problems there in my free time. Life is a bit weird sometimes.
Bootstrapping packages on a new architecture is a bit of a black art that only a fairly small number of relatively bitter and twisted people know very much about. Doing it in Ubuntu is specifically painful because we've always forbidden direct binary uploads: all binaries have to come from a build daemon. Compilers in particular often tend to be written in the language they compile, and it's not uncommon for them to build-depend on themselves: that is, you need a previous version of the compiler to build the compiler, stretching back to the dawn of time where somebody put things together with a big magnet or something. So how do you get started on a new architecture? Well, what we do in this case is we construct a binary somehow (usually involving cross-compilation) and insert it as a build-dependency for a proper build in Launchpad. The ability to do this is restricted to a small group of Canonical employees, partly because it's very easy to make mistakes and partly because things like the classic "
Reflections on Trusting Trust" are in the backs of our minds somewhere. We have an iron rule for our own sanity that the injected build-dependencies must themselves have been built from the unmodified source package in Ubuntu, although there can be source modifications further back in the chain. Fortunately, we don't need to do this very often, but it does mean that as somebody who can do it I feel an obligation to try and unblock other people where I can.
As far as constructing those build-dependencies goes, sometimes we look for binaries built by other distributions (particularly Debian), and that's pretty straightforward. In this case, though, these two architectures are pretty new and the Debian ports are only just getting going, and as far as I can tell none of the other distributions with active arm64 or ppc64el ports (or trivial name variants) has got as far as porting GHC yet. Well, OK. This was somewhere around the Christmas holidays and I had some time. Muggins here cracks his knuckles and decides to have a go at bootstrapping it from scratch. It can't be that hard, right? Not to mention that it was a blocker for over 600 entries on that build failure list I mentioned, which is definitely enough to make me sit up and take notice; we'd even had the odd customer request for it.
Several attempts later and I was starting to doubt my sanity, not least for trying in the first place. We ship GHC 7.6, and upgrading to 7.8 is not a project I'd like to tackle until the much more experienced Haskell folks in Debian have switched to it in unstable. The
porting documentation for 7.6 has bitrotted more or less beyond usability, and the
corresponding documentation for 7.8 really isn't backportable to 7.6. I tried building 7.8 for ppc64el anyway, picking that on the basis that we had quicker hardware for it and didn't seem likely to be particularly more arduous than arm64 (ho ho), and I even got to the point of having a cross-built stage2 compiler (stage1, in the cross-building case, is a GHC binary that runs on your starting architecture and generates code for your target architecture) that I could copy over to a ppc64el box and try to use as the base for a fully-native build, but it segfaulted incomprehensibly just after spawning any child process. Compilers tend to do rather a lot, especially when they're built to use GCC to generate object code, so this was a pretty serious problem, and it resisted analysis. I poked at it for a while but didn't get anywhere, and I had other things to do so declared it a write-off and gave up.
Book the Second: The golden thread of progress
In March, another mailing list conversation prodded me into finding a
blog entry by Karel Gardas on building GHC for arm64. This was enough to be worth another look, and indeed it turned out that (with some help from Karel in private mail) I was able to cross-build a compiler that actually worked and could be used to run a fully-native build that also worked. Of course this was 7.8, since as I mentioned cross-building 7.6 is unrealistically difficult unless you're considerably more of an expert on GHC's labyrinthine build system than I am. OK, no problem, right? Getting a GHC at all is the hard bit, and 7.8 must be at least as capable as 7.6, so it should be able to build 7.6 easily enough ...
Not so much. What I'd missed here was that compiler engineers generally only care very much about building the compiler with
older versions of itself, and if the language in question has any kind of deprecation cycle then the compiler itself is likely to be behind on various things compared to more typical code since it has to be buildable with older versions. This means that the removal of some deprecated interfaces from 7.8 posed a problem, as did some changes in certain
primops that had gained an associated compatibility layer in 7.8 but nobody had gone back to put the corresponding compatibility layer into 7.6. GHC supports running Haskell code through the C preprocessor, and there's a
__GLASGOW_HASKELL__ definition with the compiler's version number, so this was just a slog tracking down changes in git and adding
#ifdef-guarded code that coped with the newer compiler (remembering that stage1 will be built with 7.8 and stage2 with stage1, i.e. 7.6, from the same source tree). More inscrutably, GHC has its own packaging system called Cabal which is also used by the compiler build process to determine which subpackages to build and how to link them against each other, and some crucial subpackages weren't being built: it looked like it was stuck on picking versions from "stage0" (i.e. the initial compiler used as an input to the whole process) when it should have been building its own. Eventually I figured out that this was because GHC's use of its packaging system hadn't anticipated this case, and was selecting the higher version of the
ghc package itself from stage0 rather than the version it was about to build for itself, and thus never actually tried to build most of the compiler. Editing
ghc_stage1_DEPS in
ghc/stage1/package-data.mk after its initial generation sorted this out. One late night building round and round in circles for a while until I had something stable, and a Debian source upload to add basic support for the architecture name (and other changes which were a bit over the top in retrospect: I didn't need to touch the embedded copy of libffi, as we build with the system one), and I was able to feed this all into Launchpad and watch the builders munch away very satisfyingly at the Haskell library stack for a while.
This was all interesting, and finally all that work was actually paying off in terms of getting to watch a slew of several hundred build failures vanish from arm64 (the final count was something like 640, I think). The fly in the ointment was that ppc64el was still blocked, as the problem there wasn't building 7.6, it was getting a working 7.8. But now I really did have other much more urgent things to do, so I figured I just wouldn't get to this by release time and stuck it on the figurative shelf.
Book the Third: The track of a bug
Then, last Friday, I cleared out my urgent pile and thought I'd have another quick look. (I get a bit obsessive about things like this that smell of "interesting intellectual puzzle".) slyfox on the #ghc IRC channel gave me some general debugging advice and, particularly usefully, a reduced example program that I could use to debug just the process-spawning problem without having to wade through noise from running the rest of the compiler. I reproduced the same problem there, and then found that the program crashed earlier (in
stg_ap_0_fast, part of the run-time system) if I compiled it with
+RTS -Da -RTS. I nailed it down to a small enough region of assembly that I could see all of the assembly, the source code, and an intermediate representation or two from the compiler, and then started meditating on what makes ppc64el special.
You see, the vast majority of porting bugs come down to what I might call gross properties of the architecture. You have things like whether it's 32-bit or 64-bit, big-endian or little-endian, whether
char is signed or unsigned, that sort of thing. There's a
big table on the Debian wiki that handily summarises most of the important ones. Sometimes you have to deal with distribution-specific things like whether GL or GLES is used; often, especially for new variants of existing architectures, you have to cope with foolish configure scripts that think they can guess certain things from the architecture name and get it wrong (assuming that
powerpc* means big-endian, for instance). We often have to update
config.guess and
config.sub, and on ppc64el we have the additional hassle of updating libtool macros too. But I've done a lot of this stuff and I'd accounted for everything I could think of. ppc64el is actually a lot like amd64 in terms of many of these porting-relevant properties, and not even that far off arm64 which I'd just successfully ported GHC to, so I couldn't be dealing with anything particularly obvious. There was some hand-written assembly which certainly could have been problematic, but I'd carefully checked that this wasn't being used by the "unregisterised" (no specialised machine dependencies, so relatively easy to port but not well-optimised) build I was using. A problem around spawning processes suggested a problem with
SIGCHLD handling, but I ruled that out by slowing down the first child process that it spawned and using
strace to confirm that
SIGSEGV was the first signal received. What on earth was the problem?
From some painstaking gdb work, one thing I eventually noticed was that
stg_ap_0_fast's local stack appeared to be being corrupted by a function call, specifically a call to the colourfully-named
debugBelch. Now, when IBM's toolchain engineers were putting together ppc64el based on ppc64, they took the opportunity to fix a number of problems with their ABI: there's an
OpenJDK bug with a handy list of references. One of the things I noticed there was that there were some
stack allocation optimisations in the new ABI, which affected functions that don't call any vararg functions and don't call any functions that take enough parameters that some of them have to be passed on the stack rather than in registers.
debugBelch takes varargs: hmm. Now, the calling code isn't quite in C as such, but in a related dialect called "Cmm", a variant of C-- (yes, minus), that GHC uses to help bridge the gap between the functional world and its code generation, and which is compiled down to C by GHC. When importing C functions into Cmm, GHC generates prototypes for them, but it doesn't do enough parsing to work out the true prototype; instead, they all just get something like
extern StgFunPtr f(void);. In most architectures you can get away with this, because the arguments get passed in the usual calling convention anyway and it all works out, but on ppc64el this means that the caller doesn't generate enough stack space and then the callee tries to save its varargs onto the stack in an area that in fact belongs to the caller, and suddenly everything goes south. Things were starting to make sense.
Now,
debugBelch is only used in optional debugging code; but
runInteractiveProcess (the function associated with the initial round of failures) takes no fewer than twelve arguments, plenty to force some of them onto the stack. I poked around the GCC patch for this ABI change a bit and determined that it only optimised away the stack allocation if it had a full prototype for all the callees, so I guessed that changing those prototypes to
extern StgFunPtr f(); might work: it's still technically wrong, not least because omitting the parameter list is an obsolescent feature in C11, but it's at least just omitting information about the parameter list rather than actively lying about it. I tweaked that and ran the cross-build from scratch again. Lo and behold, suddenly I had a working compiler, and I could go through the same build-7.6-using-7.8 procedure as with arm64, much more quickly this time now that I knew what I was doing. One
upstream bug, one Debian upload, and several bootstrapping builds later, and GHC was up and running on another architecture in Launchpad. Success!
Epilogue
There's still more to do. I gather there may be a Google Summer of Code project in Linaro to write proper native code generation for GHC on arm64: this would make things a good deal faster, but also enable GHCi (the interpreter) and Template Haskell, and thus clear quite a few more build failures. Since there's already native code generation for ppc64 in GHC, getting it going for ppc64el would probably only be a couple of days' work at this point. But these are niceties by comparison, and I'm more than happy with what I got working for 14.04.
The upshot of all of this is that I may be the first non-Haskell-programmer to ever port GHC to two entirely new architectures. I'm not sure if I gain much from that personally aside from a lot of lost sleep and being considered extremely strange. It has, however, been by far the most challenging set of packages I've ported, and a fascinating trip through some odd corners of build systems and undefined behaviour that I don't normally need to touch.
This year, Sarah and I spent two weeks off the coast of Dubrovnik, on
Kolo ep island. We'd never been to Croatia before, and I was a little
nervous that if there wasn't much to do on the island, we wouldn't be
able to get elsewhere particularly easily. I needn't have worried: We
thoroughly enjoyed our stay. It might even be my favourite holiday so
far.
We did travel around a bit: twice to Dubrovnik's old town and once to
a botanical gardens in Trsteno but there was also loads to see on the
island itself. One of our favourite trips was simply to a cave on the
opposite side of the island known as a "blue cave" because you looked
like a Smurf on the inside. We had to jump into the sea from the boat
and swim into and out of the cave before continuing on our trip.
Simply puttering about on the boat, either around the island or to
Trsteno was a great experience in itself. Our guide Sammi was very
friendly, the weather was great and swimming around or just enjoying
the sun on the back of the boat was enough for me to want to check
property prices out there.
On Kolo ep island is a little restaurant called Villa
Ruza. We managed to eat there twice during
our stay. I think it is considered to be one of the top ten
restaurants in Croatia. It's one of the best places I've ever eaten
in my life. In terms of price, it was about half that of restaurants
in Dubrovnik itself.
Dubrovnik itself was, quite predictably, much busier than Kolo ep. It
is a stunningly attractive old town. On our second visit we took a
tour. There are a variety of different themed tours that you can
take, including Game Of Thrones sight-seeing tours, and there were
Game Of Thrones tat shops all over the place. We opted for A Story
About The
War.
One thing we love to do when visiting foreign places is to get an
idea of what life is like for people there, and this war was not all
that long ago - I can remember the news coverage from the time. We
were the only two people on that particular tour, so we had a fairly
tailored experience. I wish I could remember our guide's name. She
was the same age as us and grew up as a child in the city when it was
under siege. Her personal story included details about where her
family stayed; what life was like for kids growing up at the time;
how people got supplies; which areas were badly affected and much more.
It was a very heartfelt tour and gave us an intimate and personal
portrait of what life was like for people there at a time that most
tour operators tend to prefer to ignore. This was exactly what we
were looking for. Sarah and I were both a little teary at the end!
We tried to do things a little different on this holiday. Normally we
keep ourselves to ourselves and don't socialise much with other
guests. We also rarely do trips and excursions, preferring to sort
things out for ourselves. This time we made an effort to be more
sociable and I'm glad we did because we met some really nice people.
The trips we did were great fun and on the last day I blew through
most of the rest of our currency by hiring a jet ski for half an
hour. Those things are FUN. Top speed on the one I was using was
57 land mph. Every muscle in my body was screaming for a few days
afterwards!
Often on a two-week holiday we reach a point around 10 days where
we've pretty much had enough. This time around I could have stayed for
twice as long.
This was the first proper test of my new camera and I've uploaded a set of
photographs from our holiday to a Flickr
gallery.
Trsteno (near Botanical gardens)
Kolo ep bay, from Villa Ruza
A week ago, we concluded our biggest DebConf ever! It was a huge success.
Here's an odd thing about the 
The other day I received from my Japanese teacher an interesting article by Yamazaki Masakazu 
The author describes his feelings and thoughts about the shishiodoshi, in particular connects human life (stress and relieve, cycles), the flow of time, and some other concepts with the shishiodoshi. Up to here it is a wonderful article providing interesting insights into the way the author thinks. Unfortunately, then the author tries to underline his ideas by comparing the Japanese shishiodoshi with European style water fountains, describing the former with all favorable properties and full of deep meaning, while the latter is qualified as beautiful and nice, but bare of any deeper meaning.
I don t go into details that the whole comparison is anyway a bad one, as he is comparing Baroque style fountains, a very limited period, and furthermore ignores the fact that water fountains are not genuinely European (isn t there one in the Kenrokuen, one of the three most famous gardens in Japan!?), nor does he consider any other water-installation that might be available. What really destroys the in principle very nice article is the tone:
The general tone of the article then can be summarized into: The shishiodoshi is rich on meaning, connects to the life of humans, instigates philosophical reflections, represents nature, the flow of time etc. The water fountain is beautiful and gorgeous, but that is all.
I don t think that this separation, or this negative undertone, was created on purpose by the author. A person of his stature is supposedly above this level of primitive comparison. I believe that it is nothing else but a consequence of upbringing and the general attitude that permeates the whole society with this feeling of separateness.
Us and They
Repeatedly providing sentences like Japanese people and Western people have different tastes.. ( ). About 10 times in this short article expressions like Japanese and Westerner appear, leaving the reader with a bitter taste of an author that considers first the Japanese a people (what about Ainu, Ryukyu, etc?), and second that the Japanese are exclusive in the sense that they are set apart from the rest of the world in their way of thinking, living, being.
What puzzles me is that this is not only a singular opinion, but a very general straight in the Japanese media, TV, radio, newspaper, books. Everyone considers Japan and Japanese as something that is fundamentally and profoundly different from everyone else in the world.
There is We the Japanese (and that doesn t mean nationality of passport, but blood line!), and there are They the Rest or, as the way of writing and and description on many occasion suggestions, They the Barbarians .
A short anecdote will underly this: One of the favorite TV talk show / pseudo-documentary style is about Japanese living abroad. That day a lady married in Paris was interviewed. What followed was a guided tour through Paris showing: Dirt in the gutter, street cleaning cars, waste disposal places. Yes, that was all. Just about the dirt . Of course, at length the (unfortunately only apparent) cleanliness of Japanese cities and neighborhoods are mentioned and shown to remind everyone how wonderful Japan is and how dirty the Barbarians. I don t want to say that I consider Japan more dirty than most other countries just the visible part is clean, the moment you step a bit aside and around the corner, there are the worst trash just thrown away without consideration. Anyway.
To return to the topic of Us and They I consider us all humans, first and foremost, and nationality, birthplace, and all that are just by chance. I do NOT reject cultural differences, they are here, of course. But cultural differences are one thing, separating one self and one s perceived people from the rest of the world is another.
Conclusion
I repeat, I don t think that the author had any ill intentions, but it would have been nicer if the article wouldn t make such a stark distinction. He could have written about Shishiodoshi and water fountains without using the Us They categorization. He could have compared other water installations, could have discussed the long tradition of small ponds in European gardens, just to name a few things. But the author choose to highlight differences instead of commonalities.
It is the Us against Them feeling that often makes life in Japan for a foreigner difficult. Japanese are not special, Austrians, too, are not special, nor are Americans, Russians, Tibetans, or any other nationality. No nationality is special, we are all humans. Maybe at some point this will arrive also in the Japanese society and thinking.
ApologyAs a twisted turn of fate, I need to start my platform with an apology. Last year, I 
This brief announcement was released yesterday to the 
I stared at 
This is the fourth post in a series of posts (
I'm moving. Well, not right here, right now. Rather less than two years. But I already know what my flat will look like and was able to influence that decision. And there will be more to influence, like what to do with common rooms in the builing, or what to put in the garden (voting for climbing facilities for my son of course!). It's this kind of co-housing project where you already know your neighbours beforehand and can find common grounds for decisions like that.
The co-housing project I'm moving to is called 
My yoga teacher was out sick this morning. I had grand plans of instead biking
to the pool and going for a swim, but when my alarm went off, and the weather
outside was grey and miserable, having an extra half an hour lie in seemed more
attractive. I think I made the right choice, because I felt like a million
bucks today.
I did all of the preparation to bake a batch of carrot and kale muffins before
Zoe arrived, and we baked a batch as soon as she arrived and had them out of
the oven in time to drive to Playgroup. I was expecting a larger crowd today on
account of the wet weather, but it turned out quite the opposite. That said,
Zoe still had a good time. There's really no other kids her age though, so it's
still winding up as a "play with Dad in a different environment".
My ABN came through yesterday, so after Playgroup we walked to the bank to give
it to them and sort out a business credit card. Zoe was super well behaved
while I did that, so we grabbed a fresh hot cross bun from 
